Extract-Transform-Load (ETL) handles large amount of data and managesworkload through dataflows. ETL dataflows are widely regarded as complex andexpensive operations in terms of time and system resources. In order tominimize the time and the resources required by ETL dataflows, this paperpresents a framework to optimize dataflows using shared cache andparallelization techniques. The framework classifies the components in an ETLdataflow into different categories based on their data operation properties.The framework then partitions the dataflow based on the classification atdifferent granularities. Furthermore, the framework applies optimizationtechniques such as cache re-using, pipelining and multi-threading to thealready-partitioned dataflows. The proposed techniques reduce system memoryfootprint and the frequency of copying data between different components, andalso take full advantage of the computing power of multi-core processors. Theexperimental results show that the proposed optimization framework is 4.7 timesfaster than the ordinary ETL dataflows (without using the proposed optimizationtechniques), and outperforms the similar tool (Kettle).
展开▼
